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Abstract. We calculate the number of metastable states in the generalized random 
orthogonal model. The results obtained are verified by exact numerical enumeration 
for small systems sizes but taking into account finite size effects. These results are 
compared with those for Hopfield model in order to examine the effect of strict 
orthonormality of neural network patterns on the number of metastable states. 



PACS numbers: 05.20-y, 75.10 Nr 
1. Introduction 

Mean field or totally connected spin glass models are among the most widely studied 
models of complex systems. They are the starting point for understanding finite 
dimensional spin glasses and are also related to neural network models and complex 
optimization problems Such systems exhibit an exponentially large number of 
pure states and dynamic glass like transitions. Below a certain dynamic transition 
temperature the dynamics becomes very slow and the systems stay out of equilibrium 
on numerical or experimental time scales. An important factor in the slow dynamics 
is the presence of metastable or blocked configurations. The enumeration of the 
number of metastable states has been addressed by various authors in p-spin Ising 
systems with Gaussian interaction matrices 0, neural network models 0, |], random 
orthogonal models (ROMs)0 and in periodic glass models which have no quenched 
disorder 0. Recently the authors studied the statics of generalized random orthogonal 
models originally introduced in and established general criteria determining whether 
these models exhibit continuous spin glass like transitions or structural glass transitions 
preceded by a dynamical transition . The first known examples of these sorts of phase 
transitions were seen in p-spin models for p > 2 ||^. However a number of two spin Ising 
models were later shown to have this structural glass transition 0, |r^. Here 



we shall extend the results of for the ROM to a more general ROM which can be 
interpreted as a Hopfield model with strictly orthogonal patterns in order to explore 
the influence of pattern orthogonality on the number of metastable states. We also 
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analyze anti-ferromagnetic Hopfield model as this class of ROMs can also be viewed 
as anti-ferromagnetic Hopfield models. The calculation for the standard ferromagnetic 
Hopfield model was carried out in ^ and we also note that the anti-ferromagnetic 
Hopfield has a structure similar to the Hamiltonian arising in the analysis of the Nash 



equilibria in the minority game as studied in |Tl|. Our analytic results are backed up 
by exact enumeration simulations for small system sizes. Despite small system sizes we 
show that when finite size scaling is taken into account the agreement with the analytical 
results is excellent. 

The Hamiltonian in a fully connected generalized ROM is 

H = --^JijSiSj (1) 

where the Si, 1 < 1 < N are Ising spins and the interaction matrix J is statistically 
invariant under the transformation J JO where O G 0{N) (the group of 

orthogonal transformations on M^). The matrix J can thus be written as 

J = O^AO (2) 

where A is a diagonal matrix with density of eigenvalues denoted by p(A). If the support 
of p(A) is bounded on the real axis then the thermodynamic limit is well defined. It 
was shown in that the nature of the spin glass transition in such models depends 
on the behavior of the density of states p(A) in the neighborhood of Amax, the largest 
eigenvalue of A. 

In this paper we shall concentrate on the model defined by 

p(A) =a5(A- !) + (!- a)5(A + l) (3) 

The matrix J in this case may be written as 



= E ^'^j- E (4) 

{/i :A^=l} {/i :Ap=-l} 

where C,^ is a random basis of orthonormal vectors on M^. One may also write J in the 
following two forms using the completeness of the 

= 5.. -2 E (6) 

We recall that the Hopfield model with p = aN Gaussian patterns has an interaction 
matrix given by 

aN 

j^^ = Y.^e; (7) 
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Here the variables are Gaussian with zero mean and variance ^f^j* = 6ij6^^' /N. 

These patterns are only orthonormal in the statistical sense, that is J^i^i^i = ■ 
Up to a constant diagonal term, the ROM we study here is from Eq. (^) equivalent to 
a ferromagnetic Hopfield (FH) model with aN strictly orthonormal patterns, or from 
Eq. equivalent to an anti-ferromagnetic Hopfield (AFH) model with 1 — a patterns. 

The number of metastable states gives useful information about the phase space of 
complex systems. The easiest metastable states to analyze are those which are single 
spin flip stable, that is to say a configuration where flipping a single spin increases (and 
possibly leaves constant) the energy of the system. Alternatively every spin is aligned 
with its local field. A metastable state thus defined is a blocked configuration of any 
single spin flip Monte-Carlo dynamics. 



2. Average Number of Metastable States 



In this section following we explain the calculation of the number of metastable states 
for generalized ROMs. By definition the average number of metastable states is given 
by 



(8) 



The term 9 is the Heaviside function and is only nonzero if every spin Si is aligned with 
its local field hi = Xl^yj •^ij^j when hiSi > 0. The average number of metastable 
states at average energy E per spin is given by 



NMsiE) = Tis, 



(9) 



To proceed we make the gauge transformation Oij OijSiSj = O'ij, it is easy to see 
that O' is also in 0{N). One may therefore write 



NMsiE) = 2 



N 



where J' = O'^AO'. Following the standard method we use the identity 



e{x) 

We thus obtain 



dx 



d\ 
2^ 



exp(iAx) 



(10) 



(11) 



„/v / dfidxidXi 
Nms = 2 /^^exp,. 



E + ^) - ' E - ' E ^^^^ + ^^/^^ ) (12) 

ij i i / 



To simplify the algebra we make the change of variables Aj Aj — ^. Following |^ |l^ 
we now consider the term 



Q = exp(i E -^J^*) = exp(-Tr J(M - 2L)) 



(13) 
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Where in vectorial notation M = Au-^ + uA"^ with Uj = 1, Aj = Aj and L^- = (Aj— /i/2)(5jj. 
We note here that in order to ehminate the diagonal term in Eq. (|1^) the matrix L has 
appeared, and this term must be properly accounted for to obtain the correct result. 
The averaging over the 0{N) disorder (the Haar measure) in this problem can be carried 
out by using the results of for U{N) integration and adapting them to the 0{N) 
case [0]. Recently a simple replica method was also used to derive these results @. The 
results |[T^ , 1^, 1^ give for an arbitrary symmetric matrix M and J = O^AO that 

'A^ M 1 
—TrG(— )+ nonextensive terms (14) 

where the overline indicates the Haar average over O and to leading order the non 
extensive terms are of order one. A compact formula for G is 

P(A) 







exp 


-TrMJ 
2 



exp 



G{z) = max^{/i2 — dX 



A 



-\n{z) - 1} 



(15) 



In the models of interest here G{z) is given by 
1 



G{z) 



1 + 4z{m + z))^ + m In (1 + 4z{m + z))^ +2z + m 



- In ^(1 + 4z(m + 2;)) 2 + 1 + 2mz^ - m ln(m + 1) 
G{z) = -aln(l - z) (FH) 
G{z) = a\n{l + z) (AFH) 

Following the results of 0, [T^, |^ one obtains 



1 - ln(2) (ROM 



exp 



2 ^ A^ ' 



(19) 



Given the form of the matrix M, the only non zero eigenvalues of M are in the vector 
subspace of spanned by A and u . The two non zero eigenvalues are 



/i± = A ■ u ± |A| |u| 
We define the order parameters z and v by 

-TXi 



(20) 

(21) 
(22) 



Hence the matrix M = M/N has eigenvalues z + ^/v and z — which are of order 1 
and the other N — 2 eigenvalues are zero. We now consider the evaluation of the term 
TrG{i^^jr^), Taylor expanding one has 



TiGii 



M -2L. 

N ' 



5^-G(")(0)Tr 



n=0 



(M - 2L)" 



(23) 



We note that for finite values of the Aj that to leading order TrM^ = C{p)N'^ and that 
TrL*^ = D{p)N. Also for any product Pp{M,L) of the M and L containing p factors 
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one has that for p > 1 TrPp(M, L) < A^p+^ when N is large for any small positive 
e. The dominant terms of this form are when Pp{M, L) = M^. To see this, consider 
a product Pp{M, L) with at least one L occurring, we can thus write, exploiting the 
Cauchy-Schwartz inequality 

TrPp(M,L) = TrLPp_i(M,L) 

< (TrL^)^ (TrPp_i(M,L)^)^ 

< Const X iV^'+^-i = Const N^^^'^ (24) 

Hence for p >2 any product containing at least one L is such that TiPp{M,L)/N^ < 
Const A^'^~5 — i> 0. The only term containing L which survives the thermodynamic limit 
is that in the linear term of the Taylor expansion (^). Putting all this together for 
large N yields 



n = exp (^N (^^Giiz + ly^) + ]p{iz - %^) - %{z - |)G"(0) 



(25) 



It is easy to show that G'(0) = J dX Ap(A), and hence in the ROM G'{0) = 2a — 1, 
which is zero when a = 1/2 explaining why the diagonal term mentioned above 
was unimportant in the calculations of [T^. It is easy to see heuristically the 
origin of this term, as it comes from the term D = YliiJ'u'^i Eg. (p!^) , the results 
of the above calculation is to show that in the large N limit D can be written as 
^^(^E.4)(E.A.) = iTr(J)(E,A.). 

Introducing a delta function representation for the order parameters, the Xi and Aj 
and integrals may be done yielding 

A^^ /".... ^Stt 



where 



Nms{.E) = dz dv ds dt { — ] exp {NA[z, v, s, t, E]) (26) 

IGvr^ / \ t 



1 vt 
A[z, V, s, t,E] = - {G{z + iV^) + Giz - 2v^)) - zG'{0) - sz + - (27) 

+ P(4) + ^fE + ^ + ^ + iG"(0) 



where 



B(u) = In 



and anticipating a real action we have made the transformation z — > —iz The average 
energy E* of the metastable states is then given by 

= - I - ^^'(°) (29) 
at the saddle point of the reduced action 

1 l]f s 

A[z, V, s,t] = - {G{z + zv^) + G{z - zv^)) -zG'{0)-sz+-+B{^){m) 

2 2 y/t 
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The total average number of metastable states is given by 

TVm^ = N j dENMs{E) (31) 



which gives 



A^2 

Nms = — K / dz dv ds dtexp (NA[z,v, s,t]) (32) 



We have therefore to leading order 



S..HN^. ext..,„,„,4.. Ml - i^f - + 0(l/iV)(33) 

where Ti. is the Hessian of A at the saddle point and the term 0{1/N) comes from the 
nonextensive terms arising in the 0{N) disorder averaging . The fact that the leading 
order correction is 0{1/N) means that S* can be evaluated by exact enumeration for 
quite small system sizes when the above finite size scaling is taken into account. 

The extremization of this action (with four order parameters) seems quite 
complicated and has a similar structure to the saddle point encountered in the 
calculation of the average number of metastable states in the periodic glass model 
studied in § the Hopfield model |, || and the ROM at a = 1/2 §, ig. The saddle 
point equations giving s and t are 

s=^G'iz + iV^) + ^G'{z-iy/^)-G\0) (34) 
t = - ^{G'iz + zv^) - G'{z - ^^/^)) (35) 

The remaining saddle point equations must be solved numerically. Given the definition 
of the order parameter v we look for solutions to the saddle point equations with positive 
V. We shall see that the solutions we find with this prescription agree perfectly with 
the results of exact enumeration of small systems where we can calculate both S* the 
entropy of metastable states and E* the average energy of these states. 

The calculated values of S*{a) for the ROM and ferromagnetic Hopfield models 
along with S*{1 — a) for the anti-ferromagnetic Hopfield model are shown in Fig. (|l]). 
As the ROM can be regarded as an anti-ferromagnetic Hopfield model with 1 — a 
orthonormal patterns or a ferromagnetic Hopfield model with a orthonormal patterns, 
this comparison is natural. In the limit a —>■ 1 we see in Fig. (|l]) that >S'^om('^) ~^ 
5'Xpjj(1 — a) and that in these two cases S* — > ln(2). However 5*^^:^ ~ 0.131486. Hence, 
for the ROM, as a — >■ 1 there is a small fraction (1— «) of repulsive patterns to be avoided 
to minimize the energy and the fact that they are strictly orthonormal or statistically 
orthonormal does not change the behavior of S* drastically with respect to the AFH. 
This result can be seen analytically as follows. Writing a' = 1 — a, then near a = 1, for 
the ROM 

G{z) = z-a' ln(l + 2z) + 0{a'^) (36) 

In the metastable state calculations here the term linear in z of G{z) disappears from 
the calculation. The remaining term is just (up to a rescaling of the energy that will not 
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Figure 1. The entropy of metastable states per spin for (i) ROM (full line) (ii) FH 
with p — aN patterns (dotted line) and (iii) AFH with p — {l — a)N patterns (dashed 
line) 



affect ttie number of metastable states) tlie term one lias for tlie AFH witli a' patterns. 
Tlius explaining the convergence of S*{a) for the ROM with 5'*(1 — a) for the AFH 
near a = 1. One may further show that in these two cases as a — 1 one has 

1 



S*{a) 



M2)-| 



In 



ea' a' 



+ 



(37) 



This asymptotic formula agrees well with the numerically calculated value up to a' = 0.1. 

In the limit a — > we also see that 5'^om(0^) — ^Fni^~^) — t)ut 
'S'afh(I) ~ 0.306983. However S'^qm ^fh remain different as a 0. Hence when 
the attractive patterns are strictly orthonormal then there are more metastable states 
than if the patterns are only statistically orthonormal. In the FH model in the limit 
a — > it was shown that S*{a) ^ ^[ln(2/7ra) — 1]. In the same limit in the ROM 
the asymptotic behavior is rather singular and we have not yet found the corresponding 
asymptotic behavior. 



To summarize we have the inequality Sp 



FH 



a, 



a 



Hence 



in the case of ferromagnetic Hopfield models strict pattern orthonormality increases the 
number of metastable states but in the anti-ferromagnetic Hopfield model it decreases 
the number of metastable states. Let us note here that at a fixed pattern number the 
ROM and AFH have more metastable states than the FH. This is in accordance with 
the observation that the ROM and AFH have a structural glass transition whereas the 
FH has a continuous spin glass transition. 
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3. Numerical Simulations 

To verify our results we have carried out exact numerical enumeration of Nms on small 
systems of size between 10 and 30 spins. Using the finite size scaling predicted by Eq. 
(^) we find excellent agreement between the calculations presented here even for the 
relatively small system sizes examined. 

Measured in the simulations were Nms, ln(A^A/5) and E* the average value of the 
energy per spin of the metastable states. The numerical results confirm to high precision 
that hi{NMs) = ln(A^Afs)5 thus confirming that the total entropy on metastable states is 
self averaging and justifying our annealed calculation. Averaging was carried out over 
up to 2^^~^ samples for the systems size of size N , with N between 10 and 30. Both 
the annealed total entropy of metastable states NS* = Iji^Nms) and the quenched total 
entropy NS*\n{NMs) were plotted as function of and were found to be very close to 
straight lines for systems of size greater than 10. The value of 5** was then determined 
by a linear fit. The average energy per spin over all metastable states and samples, 
corresponding to the annealed calculations carried out here, 

E' = /''^^^W^ (38) 
JdENMs(E) 

was calculated from the simulations of the systems of size 20 (to have good statistics). 
As an additional check the quenched average energy per spin of the metastable states 

( JdE ENmsW) 
V JdENMsiE) 

was also calculated. We note that if annealed approximation is exact then we should 
find E* = E*. 

The results for S* estimated from the annealed average \n{NMs) and the quenched 
average ln{NMs) in the AFH are shown in Fig. (|^) against the calculated value. We 
see again that the agreement is excellent. The difference between the annealed and 
quenched averages are very small showing the validity of the annealed approximation. 
Note that by Jensen's inequality the the annealed average should be greater than the 
quenched one. Similarly the numerically estimated values for E* and E* are within the 
error bars of the simulation and also in excellent agreement with the calculated value of 
E*. 

For the ROM, in each system of size A^, Na eigenvectors were chosen to have 
eigenvalue 1 and the remaining to have eigenvalue —1. The extrapolated values of S* 
and S* are shown in Fig.(^ along with the energy E* obtained from samples of size 
A^ = 30. We see that again that agreement with the analytical calculations is excellent 
and that the extrapolated values of S* and 5"* coincide. 

Via the exact numerical enumeration were also computed the annealed NS*{E) = 
\n{NMs{E)) and quenched NS*{E) = \n{NMs{E)) entropy of the metastable states of 
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Figure 2. The annealed {S*) (circles) and quenched (S*) (squares) entropy of 
metastable states for the anti-ferromagnetic Hopfield model measured from the 
simulations along with the calculated value (solid line). Also shown is the average 
values of the energy per spin of the metastable states measured by the simulations 
(diamonds) along with the calculated value E* . 



0.7 



0.5 



0.3 



•lu 0.1 

T3 
CO 

w -0.1 



-0.3 
-0.5 
-0.7 



0.2 0.4 0.6 0.! 

a 



Figure 3. As in Fig. but for the generalized ROM. For the average energy, the 
errors where computed and are always less than the size of the symbols. 




Figure 4. The annealed {S*{E)) (circles, solid line) and quenched {S*{E)) (squares, 
dotted line) entropy of metastable states of energy E per spin for the ROM. 



energy E. Numerically, we computed 

i^samples ^ AE 
1=1 



NusiE,) - „,^^ 2^ j-j^ (40) 



and 



'''^'''^'^''^^^ = W^^^s g ) ^ ^ 

where AE = .005 is the chosen bin size of the discrete energy values Ei. To avoid 
divergences when taking the logarithm, we set \n{NMs{E)) = when Nms{E) was zero 
in a given sample. This procedure should be unimportant in the thermodynamic limit 
as it concerns a nonextensive number of metastable states. The results for the ROM 
with a = 1/2 are shown in Fig.(^) for an averaging over 32 samples of size 30. We 
see clearly, for a substantial region around the most probable energy, the annealed and 
quenched entropies coincide i.e. S*{E) = S*{E). For values of the entropy smaller than 
0.3, the two curves depart from each other, with 5** always smaller than S* as it should 
be. However, the small number of metastable states considered for these energies and 
size makes it impossible to draw any conclusion about the thermodynamic limit. In 
particular one would expect that the two entropies should collapse for all energies above 
some energy threshold Eq below E*. This is not the case for our data indicating strong 
finite size effects at the edges of the energy spectrum of the metastable states, as one 
should expect. 
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4. Conclusions 

We have shown that in the class of ROMs considered here that there is always 
an exponentially large number of metastable states which increases as a function 
of a. The ROM can be viewed as a ferromagnetic Hopfield model with a strictly 
orthonormal patterns or and anti-ferromagnetic Hopfield model with 1 — a strictly 
orthonormal patterns. Comparison with the corresponding Hopfield models shows that 
the orthonormality of the patterns increases the number of metastable states in the 
ferromagnetic case but decreases this number in the anti-ferromagnetic case. If one 
considers a ferromagnetic Hopfield model with all patterns parallel then there are clearly 
only two metastable states (all spins aligned or anti-aligned with this pattern). In the 
anti-ferromagnetic Hopfield model if all the patterns are parallel then there are more 
metastable states as it is easier to be orthogonal to a single pattern than several. This 
reasoning in these extreme cases is compatible with the results found here. 

Finally the numerical simulations carried out here, though for small system sizes, 
show remarkable agreement with the analytic calculations carried out here. This is 
extremely important as the structure of the saddle point equations is so complicated that 
one needs some confirmation that one has found the good saddle point. Furthermore 
it suggests that rather than doing Monte Carlo simulations for systems exhibiting a 
dynamical transition, where even small size systems will stay out of equilibrium, it may 
be more useful to carry out exact enumeration on these small system sizes to calculate 
thermodynamic quantities. 
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